Generating Reproducible Statistical Analyses and Evaluation Reports: Principles, Practices, and Free Software Tools

Demonstration at Evaluation 2024: Amplifying and Empowering Voices, annual conference of the American Evaluation Association, Portland, OR

Steven J. Pierce

Center for Statistical Training and Consulting, Michigan State University

2024-10-22

Outline

What do you mean by reproducible?
Why should we aim for reproducibility?
How do we achieve reproducibility?
Live Demonstration

What do you mean by
reproducible?

Let’s define some concepts.

Reproducible Research (RR)

… is achieved when investigators share all the materials required to exactly recreate the findings so that others can verify them or conduct alternative analyses.

Statistical Results Can Be:

Repeatable

Original analyst
Original data

Reproducible

New analyst
Original data

Replicable

New analyst
New data

RR is a Product

RR is a product of how we work, not which methods we use.

Why aim for
reproducibility?

Several forces are promoting and enabling the push toward reproducibility.

Methods Matter!

Irreproducible < Reproducible < Replicated

Important

Reproducibility is an attainable minimum standard for science^[1].

Remember that our methods matter: Studies that use good scientific practices yield more trustworthy, credible findings. Let’s link this continuum to reproducibility. >

At the low end, we have irreproducible findings, which rely on investigator reputation to establish credibility because key methodological details aren’t published and the findings can’t be independently verified. This limits our ability to judge their quality and provides a poor foundation for future replication studies.

Reproducible findings can be independently verified from the original data, reducing reliance on investigator reputation. We can better judge the quality of such research on its own merits. Furthermore, it provides methodological details that lay a solid foundation for future replications.

Finally, replicated research has the highest level of credibility because independent investigators have replicated the findings with new studies, indicating that the probability of opportunistic biases is low and the findings generalize beyond the original sample. This dramatically increases trustworthiness of a finding. >

Roger Peng has argued that RR is an attainable minimum standard for science (Peng et al., 2006).

Guiding Principles for Evaluators

Pursuing reproducibility enacts our guiding principles^[2]:

AEA logo.

Systematic inquiry
Competence
Integrity

Funders Value RR

Data sharing and reproducibility initiatives

NIH logo. NSF logo.

IES logo.

Emerging Scientific Norms

Open science:
- https://www.cos.io/open-science
- https://open.science.gov
Pre-registered methods^[3–5]
Reporting guidelines^[6–9]
Publishing raw data^[10,11]
Journal & publisher policies^[4,10–13]
Institutional policies

Publishing Technology

Career Benefits of RR

Motivation to focus on quality
Become more efficient
Create more products
Easier to get published
Get cited more often
Build your reputation

How do we achieve reproducibility?

Understand the criteria, then apply principles, practices, & tools.

Criteria for Reproducbility
^[1]

Data
Methods
Documentation
Distribution

Materials Required to Recreate Findings

Materials	Findings
Manuals & procedures	Statistics
Instruments & scoring rules	Coefficients & p-values
Codebooks	Confidence intervals
Methods applied	Effect sizes
Data mgt decisions	Model fit indices
Data files	Figures
Software & analysis scripts	Tables

Principles for Achieving Reproducibility

Collaboration
Organization
Automation

Preservation
Integration
Separation

Understanding a few guiding principles helps you figure out how to enhance reproducibility. Here are the principles I’ve identified so far. Thinking about reproducibility as collaboration encourages us to prepare and share materials that will make our work accessible to others. Organization simplifies communicating to others how to reproduce our work. Automation helps us achieve efficiency and accuracy because scripts can be faster and more reliable than manual processes. The preservation principle guides us to keep our raw data pristine and to keep a good audit trail. The integration principle suggests building documentation directly into data files, putting comments in code, and using code to directly insert statistical results into reports. The separation principle helps us manage complexity by splitting projects or tasks into pieces; for example separating data management from analysis, or keeping raw data separate from working data files used for specific analyses. Keep an eye out for instances of these principles in action throughout the rest of this talk.

Workflow Woes

Use GUI, menus, & dialog boxes to do tasks
Manually updating data files
Version control through saving to new file names
Disorganized folders & files
No audit trail
Copy & paste output from stats software to word processor
Fixing mistakes took lots of time

I’m going to describe some common workflow woes that may soound uncomfortably familiar to you from your own experience. As a student, I was taught a manual workflow for doing data analysis and writing reports. We used the GUI, menus, and dialog boxes in SPSS to do data management and analysis. We updated data files manually. We created tons of files, saving to a new filename as changes accrued. That led to confusing proliferation of folders and files with little planned structure. It was hard to tell who had last edited various files and which versions to use. We retyped or pasted results from SPSS output into Word documents then spent lots of time formatting them. If we made a mistake, it could takes hours or even days to re-do things. Over time I learned some strategies to improve that workflow, such as using SPSS syntax files to automate tasks, adding comments in the syntax files, adopting file naming conventions, and so on. But part of the problem was that the software tools weren’t integrated and lacked support for better workflows. Technology has vastly improved since my student days, but many evaluators still use manual workflows like those I just described. They don’t know what is possible now. Given what I’ve learned in recent years, if asked to resume using those older tools and workflows, I’d probably break out my favorite Spock quote.

Spock

“Captain, you’re asking me to work with equipment which is hardly very far ahead of stone knives and bearskins.”

Star Trek (1966) - S01E28 The City on the Edge of Forever

Free, Open Source Software Tools for Reproducible Research

R^[14], for statistical computing
RStudio Desktop^[15], an editor
Quarto^[16], for dynamic documents
Git^[17], for version control
GitHub.com, for sharing & collaborating on code

Dynamic Documents Via Quarto + R

Dynamic documents are a crucial technology for reproducibility. They implement the concept of literate programming, where one embeds code directly in a document such as a report, interleaving it with narrative text, headings, and in-text citations. When you render dynamic documents, the code is executed and the results are inserted into the output. That includes statistics computed from the data, figures, tables, formatted equations, fully-formatted reference sections, and more. This diagram shows the files involved for a simple example. The idea is that rendering your report script produces an output such as a PDF file that is ready for delivery to your client.

Several principles are in action here: Separation of data from code, integration of R code into the Quarto script, where it automates data management and analyses. Quarto automates inserting results into the report. The BibTeX file contains data about references you cite in the report, while the CSL file tells Quarto how to format the citations and the reference list. I’m using similar file names to stay organized: they show which script creates the report. Saving all the files follows the preservation principle. One can rapidly reproduce the report by re-rendering the script, or fix an error in the script then render again to automate updating the report.

Research Compendium

A compendium organizes digital files so others can review or use them to reproduce results, or do new analyses.^[18] It should:

Use prevailing conventions to organize the project
Separate data, method, & output but show how they are related
Specify the software used in the original analysis

R Packages

… are folders of digital files designed for sharing code & help documentation.^[19] They

Organize files into a conventional structure
Can contain data, meta-data, & other documentation
Can contain dynamic documents & rendered output

Tip

Use an R package for a research compendium!^[18]

Git Repositories

… are folders of files being tracked by Git^[17] for version control purposes.^[20,21] They:

Preserve a history of changes to each tracked file
Allow recovering a file’s prior state from the history
Can be local or remote (hosted on a server)
Can be private or public

Tip

Put your research compendium in a Git repository!

Git is software for doing version control on code and other files stored in repositories, which are just–you guessed it–digital folders. You can choose which files to track, then log of all committed changes to them, when they were made, who made them, and why. That preserves a fantastic audit trail. You can recover the exact state of a file as it existed at any commit in the history. You can make repositories either private or public. I usually keep a repository private until the corresponding paper has been published, then I can make it public to enhance reproducibility. So, putting your compendium in a Git repository both follows some of the principles I’ve highlighted and helps solve some of those worklflow woes I described earlier. For example, you can stop using long, confusing filenames as a crude form of version control. Give each file a simple name that stays constant then let Git track how it changes over time. Jenny Bryan’s paper on using proper version control persuaded me to start using Git. I have not regretted it! > So, put your compendium in a Git repository!

GitHub

While Git handles preservation, GitHub solves other workflow problems. GitHub is an online server that hosts Git repositories, allowing multiple people to clone copies from the remote server to their own computers and synchronize their local copies with the main one hosted on GitHub. This facilitates collaboration because each person can independently edit files in their own local copy, pull the recent changes others have made down from GitHub, and push their own changes back up to the main repository. Git will detect conflicting changes and force a user to resolve them and commit the result.

Using a private GitHub repository while working on the project allows you collaborate with teammates as you develop a report. Later, you can make the repository public so that people outside your team can access reproducibility materials.

RStudio Projects

… are folders of files that:

Keep files for an analysis project together
Can have subfolders for organizing files
Serve as R working directories w/ workspace & history
Store settings in an RStudio project file (*.Rproj)

Tip

Turn your compendium into an RStudio project!

RStudio User Interface

R Console pane
R Packages pane
R Environment pane
Script editor has a Quarto render button
Viewer & Presentation panes for rendered output
Git pane

Quarto Projects

… are folders of files that allow:

Rendering some or all dynamic documents
Sharing a configuration file across documents
Redirecting output to another folder
Option to freeze rendered output unless the code has changed

Tip

Turn a subfolder of your compendium into a Quarto project!

Example Folder Structure

MyStudy/          [Compendium, Git repository, R package, RStudio project]
  - .git/           [Hidden folder, holds Git tracking database]
  - .Rproj.user/    [Hidden folder, holds RStudio temporary files]
  - data/           [Holds R data files created by scripts]
  - man/            [Holds R help files for package & custom functions]
  - R/              [Holds R scripts w/ custom functions]
  - scripts/        [Quarto project, holds dynamic documents]  
    - extdata/      [Holds external data files to be imported]
    - output/       [Holds rendered output]
  - .gitignore      [Tells Git what to omit from tracking]
  - DESCRIPTION     [R package meta-data]
  - MyProject.Rproj [RStudio project file & settings]
  - NEWS.md         [News for users re: changes to package]
  - README.Rmd      [Dynamic document, creates README.md]
  - README.md       [Rendered output, R package documentation]

Suppose I’ve got a project creatively named MyStudy. Here’s what its folder structure might look like. The MyStudy/ folder is a research compendium that is also a Git repository, an R package, and an RStudio project. It is organized with several subfolders that help separate R data files from external data in other software formats that must be imported before use. It also separates data from code for either custom functions (the R subfolder) or for dynamic documents that are in a scripts subfolder that is also a Quarto project. Much of this structure is driven by R package conventions. Notice that output is also separated from the code by tucking it in its own subfolder. There are a handful of files for storing package meta-data and documentation, controlling what Git will track and what it will ignore. This basic structure has been working very well for me across several projects.

Collaboration Stakeholders

Your future self
Your team
Peer reviewers
Your audience

Organization

Organize files into a research compendium.
Create documentation (README files, vignettes, etc.).
Use folder structures, naming conventions,
Use data flow diagrams and/or rendering scripts

Automation

Use scripts instead of GUIs
Make computers do the tedious labor
Computers are patient idiots
Make scripts that start from a clean workspace
Test scripts frequently
Render entire scripts in a clean R session

Preservation

What should you preserve?

Data and meta-data
Methodology decisions
Data cleaning, management, and analysis scripts
Key decisions and rationales
Source code for scripts
Version histories of data, code, output, and deliverables
Who changed what & when
Software environment & versions used

Integration

Attach meta-data directly to data
Include codebooks (or scripts to generate them from the data)
Integrate narrative text with code via Markup languages
Add custom functions and associated help files to research compendium
Include references

Separation

Data from code
Raw data from cleaned data
Code from output
Draft from production output
Version history from current files

Version Control

Git tracks how files changed (additions & deletions), who changed them, when, & why
Searchable & recoverable version history
Eliminates the need to use filename conventions for versioning

Live Demonstration

Let’s take a look at actual dynamic documents!

MyStudy Files

Quarto scripts
- scripts/Create_MyStudy_Data.qmd
- scripts/MyStudy_Report.qmd
Rendered output
- scripts/output/Create_MyStudy_Data.html
- scripts/output/MyStudy_Report.html

Narrative Text

Formatting
Headings
Links

Inline R Code

Tables

Figures

Parameterized Reports

Map Your Path to Reproducibility

References

1. Peng, R. D., Dominici, F., & Zeger, S. L. (2006). Reproducible epidemiologic research. American Journal of Epidemiology, 163(9), 783–789. https://doi.org/10.1093/aje/kwj093

2. American Evaluation Association. (2018). Guiding principles for evaluators [Web Page]. Author. https://www.eval.org/About/Guiding-Principles

3. Bosnjak, M., Fiebach, C. J., Mellor, D., Mueller, S., O’Connor, D. B., Oswald, F. L., & Sokol, R. I. (2022). A template for preregistration of quantitative research in psychology: Report of the Joint Psychological Societies Preregistration Task Force. American Psychologist, 77(4), 602–615. https://doi.org/10.1037/amp0000879

4. DeCoster, J., Sparks, E. A., Sparks, J. C., Sparks, G. G., & Sparks, C. W. (2015). Opportunistic biases: Their origins, effects, and an integrated solution. American Psychologist, 70(6), 499–514. https://doi.org/10.1037/a0039191

5. Moore, D. A. (2016). Preregister if you want to. American Psychologist, 71(3), 238–239. https://doi.org/10.1037/a0040195

6. Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M. (2018). Journal article reporting standards for quantitative research in psychology: The APA publications and communications board task force report. American Psychologist, 73(1), 3–25. https://doi.org/10.1037/amp0000191

7. Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gøtzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010). CONSORT 2010 explanation and elaboration: Updated guidelines for reporting parallel group randomized trials. Journal of Clinical Epidemiology, 63(8), e1–e37. https://doi.org/10.1016/j.jclinepi.2010.03.004

8. Schulz, K. F., Altman, D. G., & Moher, D. (2010). CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials. Journal of Clinical Epidemiology, 63(8), 834–840. https://doi.org/10.1016/j.jclinepi.2010.02.005

9. von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007). The strengthening the reporting of observational studies in epidemiology (STROBE) statement: Guidelines for reporting observational studies. PLoS Medicine, 4(10), e296. https://doi.org/10.1371/journal.pmed.0040296

10. Hrynaszkiewicz, I., & Altman, D. G. (2009). Towards agreement on best practice for publishing raw clinical trial data. Trials, 10(1), 17. https://doi.org/10.1186/1745-6215-10-17

11. Hrynaszkiewicz, I., Norton, M. L., Vickers, A. J., & Altman, D. G. (2010). Preparing raw clinical data for publication: Guidance for journal editors, authors, and peer reviewers. Trials, 11(1), 9. https://doi.org/10.1186/1745-6215-11-9

12. Laine, C., Goodman, S. N., Griswold, M. E., & Sox, H. C. (2007). Reproducible research: Moving toward research the public can really trust. Annals of Internal Medicine, 146(6), 450–453. https://doi.org/10.7326/0003-4819-146-6-200703200-00154

13. Peng, R. D. (2009). Reproducible research and biostatistics. Biostatistics, 10(3), 405–408. https://doi.org/10.1093/biostatistics/kxp014

14. R Development Core Team. (2024). R: A language and environment for statistical computing (Version 4.4.1) [Computer Program]. R Foundation for Statistical Computing. http://www.R-project.org.

15. RStudio Team. (2024). RStudio Desktop: Integrated development environment for R (Version 2024.04.2+764) [Computer Program]. Posit Software, PBC. https://posit.co

16. Allaire, J. J., Dervieux, C., Scheidegger, C., Teague, C., & Xie, Y. (2024). Quarto (Version 1.5.57) [Computer Program]. Posit Software, PBC. https://quarto.org

17. Torvalds, L., Hamano, J. C., & other contributors to the Git Project. (2024). Git for Windows (Version 2.46.2(1)) [Computer Program]. Software Freedom Conservancy. https://git-scm.com

18. Marwick, B., Boettiger, C., & Mullen, L. (2018). Packaging data analytical work reproducibly using r (and friends). The American Statistician, 72(1), 80–88. https://doi.org/10.1080/00031305.2017.1375986

19. Wickham, H., & Bryan, J. (2021). R packages: Organize, test, document, and share your code. O’Reilly Media. https://r-pkgs.org

20. Bryan, J. (2018). Excuse me, do you have a moment to talk about version control? The American Statistician, 72(1), 20–27. https://doi.org/10.1080/00031305.2017.1399928

21. Chacon, S., & Straub, B. (2014). Pro Git. Apress Media. https://git-scm.com/book/en/v2

Generating Reproducible Statistical Analyses and Evaluation Reports: Principles, Practices, and Free Software Tools

Outline

What do you mean by reproducible?

Reproducible Research (RR)

Statistical Results Can Be:

RR is a Product

Why aim for reproducibility?

Methods Matter!

Guiding Principles for Evaluators

Funders Value RR

Emerging Scientific Norms

Publishing Technology

Career Benefits of RR

How do we achieve reproducibility?

Criteria for Reproducbility [1]

Materials Required to Recreate Findings

Principles for Achieving Reproducibility

Workflow Woes

Spock

Free, Open Source Software Tools for Reproducible Research

Dynamic Documents Via Quarto + R

Research Compendium

R Packages

Git Repositories

GitHub

RStudio Projects

RStudio User Interface

Quarto Projects

Example Folder Structure

Collaboration Stakeholders

Organization

Automation

Preservation

Integration

Separation

Version Control

Live Demonstration

MyStudy Files

Narrative Text

Inline R Code

Tables

Figures

Parameterized Reports

Map Your Path to Reproducibility

References

What do you mean by
reproducible?

Why aim for
reproducibility?

Criteria for Reproducbility
^[1]